{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ea0a577e",
   "metadata": {},
   "source": [
    "# Find Noisy Labels in Regression Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e15b9f2f",
   "metadata": {},
   "source": [
    "This 5-minute quickstart tutorial uses cleanlab to find potentially incorrect numeric values in a dataset column by means of a regression model. Unlike classification models, regression predicts numeric quantities such as price, income, age,... Response values in regression datasets may be corrupted due to: data entry or measurement errors, noise from sensors or other processes, or broken data pipelines. To find corrupted values in a numeric column, we treat it as the target value, i.e. label, to be predicted by a regression model and then use cleanlab to decide when the model predictions are trustworthy while deviating from the observed label value.\n",
    "\n",
    "In this tutorial, we consider a student grades dataset, which records three exam grades and some optional notes for over 900 students, each being assigned a final score. Combined with any regression model of your choosing, cleanlab automatically identifies examples in this dataset that have incorrect final scores.\n",
    "\n",
    "**Overview of what we’ll do in this tutorial:**\n",
    "\n",
    "- Fit a simple Gradient Boosting model (any other model could be used) on the exam-score and notes (covariates) in order to compute out-of-sample predictions of the final grade (the response variable in our regression).\n",
    "- Use cleanlab's `CleanLearning.find_label_issues()` method to identify potentially incorrect final grade values based on outputs from this regression model.\n",
    "- Train a more robust version of the same model after dropping the identified label errors using CleanLearning.\n",
    "- Run an alternative workflow to detect errors via cleanlab's `Datalab` audit, which can simultaneously estimate **many other types of data issues**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612a355a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "Quickstart\n",
    "<br/>\n",
    "    \n",
    "Already have an sklearn-compatible regression `model`, features/covariates `X`, and a label/target variable `y`? Run the code below to train your `model` and identify potentially incorrect `y` values in your dataset.\n",
    "\n",
    "\n",
    "<div  class=markdown markdown=\"1\" style=\"background:white;margin:16px\">  \n",
    "    \n",
    "```python\n",
    "\n",
    "from cleanlab.regression.learn import CleanLearning\n",
    "\n",
    "cl = CleanLearning(model)\n",
    "cl.fit(X, y)\n",
    "label_issues = cl.get_label_issues()\n",
    "preds = cl.predict(X_test)  # predictions from a version of your model trained on auto-cleaned data\n",
    "```\n",
    "    \n",
    "</div>\n",
    "    \n",
    "Is your model/data not compatible with `CleanLearning`? You can instead run cross-validation on your model to get out-of-sample `predictions`. With that, run the code below to find data and label issues in your regression dataset:\n",
    "\n",
    "<div  class=markdown markdown=\"1\" style=\"background:white;margin:16px\">  \n",
    "    \n",
    "```python\n",
    "\n",
    "from cleanlab import Datalab\n",
    "\n",
    "# Assuming your dataset has a label column named 'label'\n",
    "lab = Datalab(dataset, label_name='label', task='regression')\n",
    "# To detect more data issue types, optionally supply `features` (numeric dataset values or model embeddings of the data)\n",
    "lab.find_issues(pred_probs=predictions, features=features)\n",
    "\n",
    "lab.report()\n",
    "   \n",
    "```\n",
    "    \n",
    "</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9a290d6",
   "metadata": {},
   "source": [
    "## 1. Install required dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8430ca39",
   "metadata": {},
   "source": [
    "You can use `pip` to install all packages required for this tutorial as follows:\n",
    "\n",
    "```ipython3\n",
    "!pip install matplotlib\n",
    "!pip install cleanlab[datalab]\n",
    "# Make sure to install the version corresponding to this tutorial\n",
    "# E.g. if viewing master branch documentation:\n",
    "#     !pip install git+https://github.com/cleanlab/cleanlab.git\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e1af7d8",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# Package installation (hidden on docs website).\n",
    "\n",
    "dependencies = [\"cleanlab\", \"matplotlib>=3.6.0\", \"datasets\"]\n",
    "\n",
    "if \"google.colab\" in str(get_ipython()):  # Check if it's running in Google Colab\n",
    "    %pip install cleanlab  # for colab\n",
    "    cmd = \" \".join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
    "    %pip install $cmd\n",
    "else:\n",
    "    dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
    "                         else dependency.split('<')[0] if '<' in dependency \n",
    "                         else dependency.split('=')[0] for dependency in dependencies]\n",
    "    missing_dependencies = []\n",
    "    for dependency in dependencies_test:\n",
    "        try:\n",
    "            __import__(dependency)\n",
    "        except ImportError:\n",
    "            missing_dependencies.append(dependency)\n",
    "\n",
    "    if len(missing_dependencies) > 0:\n",
    "        print(\"Missing required dependencies:\")\n",
    "        print(*missing_dependencies, sep=\", \")\n",
    "        print(\"\\nPlease install them before running the rest of this notebook.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fb10b8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.ensemble import HistGradientBoostingRegressor\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.metrics import r2_score\n",
    "\n",
    "from cleanlab.regression.learn import CleanLearning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "284dc264",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# This cell is hidden from docs.cleanlab.ai \n",
    "\n",
    "import random \n",
    "import numpy as np \n",
    "\n",
    "SEED = 111 # for reproducibility \n",
    "\n",
    "np.random.seed(SEED)\n",
    "random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2035042e",
   "metadata": {},
   "source": [
    "## 2. Load and process the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f7450db",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv(\"https://s.cleanlab.ai/student_grades_r/train.csv\")\n",
    "test_data = pd.read_csv(\"https://s.cleanlab.ai/student_grades_r/test.csv\")\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa0165ef",
   "metadata": {},
   "source": [
    "In the DataFrame above, `final_score` represents the noisy scores and `true_final_score` represents the ground truth. Note that ground truth is usually not available in real-world datasets, and is just added in this tutorial dataset for demonstration purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82285102",
   "metadata": {},
   "source": [
    "We show a 3D scatter plot of the exam grades, with the color hue corresponding to the final score for each student. Incorrect datapoints are marked with an **X**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8173840",
   "metadata": {},
   "source": [
    "<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
    "    \n",
    "```ipython3\n",
    "# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
    "    \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "def plot_data(train_data, errors_idx):\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111, projection='3d')\n",
    "\n",
    "    x, y, z = train_data[\"exam_1\"], train_data[\"exam_2\"], train_data[\"exam_3\"]\n",
    "    labels = train_data[\"final_score\"]\n",
    "\n",
    "    img = ax.scatter(x, y, z, c=labels, cmap=\"jet\")\n",
    "    fig.colorbar(img)\n",
    "\n",
    "    ax.plot(\n",
    "        x.iloc[errors_idx],\n",
    "        y.iloc[errors_idx],\n",
    "        z.iloc[errors_idx],\n",
    "        \"x\",\n",
    "        markeredgecolor=\"black\",\n",
    "        markersize=10,\n",
    "        markeredgewidth=2.5,\n",
    "        alpha=0.8,\n",
    "        label=\"Label Errors\"\n",
    "    )\n",
    "    ax.legend()\n",
    "```\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55513fed",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "def plot_data(train_data, errors_idx):\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111, projection='3d')\n",
    "\n",
    "    x, y, z = train_data[\"exam_1\"], train_data[\"exam_2\"], train_data[\"exam_3\"]\n",
    "    labels = train_data[\"final_score\"]\n",
    "\n",
    "    img = ax.scatter(x, y, z, c=labels, cmap=\"jet\")\n",
    "    fig.colorbar(img)\n",
    "\n",
    "    ax.plot(\n",
    "        x.iloc[errors_idx],\n",
    "        y.iloc[errors_idx],\n",
    "        z.iloc[errors_idx],\n",
    "        \"x\",\n",
    "        markeredgecolor=\"black\",\n",
    "        markersize=10,\n",
    "        markeredgewidth=2.5,\n",
    "        alpha=0.8,\n",
    "        label=\"Label Errors\"\n",
    "    )\n",
    "    ax.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df5a0f59",
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_mask = train_data[\"final_score\"] != train_data[\"true_final_score\"]\n",
    "errors_idx = np.where(errors_mask == 1)\n",
    "\n",
    "plot_data(train_data, errors_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add939ae",
   "metadata": {},
   "source": [
    "Next we preprocess the data by applying one-hot encoding to features with categorical data (this is optional if your regression model can work directly with categorical features)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7af78a8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_columns = [\"exam_1\", \"exam_2\", \"exam_3\", \"notes\"]\n",
    "predicted_column = \"final_score\"\n",
    "\n",
    "X_train_raw, y_train = train_data[feature_columns], train_data[predicted_column]\n",
    "X_test_raw, y_test = test_data[feature_columns], test_data[predicted_column]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9556c624",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = [\"notes\"]\n",
    "X_train = pd.get_dummies(X_train_raw, columns=categorical_features)\n",
    "X_test = pd.get_dummies(X_test_raw, columns=categorical_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce924cf",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "Bringing Your Own Data (BYOD)?\n",
    "\n",
    "Assign your data's features to variable `X` and the target values to variable `y` instead, then continue with the rest of the tutorial.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b14309d",
   "metadata": {},
   "source": [
    "## 3. Define a regression model and use cleanlab to find potential label errors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81ee2349",
   "metadata": {},
   "source": [
    "We'll first demonstrate regression with noisy labels via the `CleanLearning` class that can wrap any scikit-learn compatible regression model you have. `CleanLearning` uses your model to estimate label issues (i.e. noisy `y`-values) and train a more robust version of the same model when the original data contains noisy labels.\n",
    "\n",
    "Here we define a `CleanLearning` object with a histogram-based gradient boosting model (sklearn version of XGBoost) and use the `find_label_issues` method to find potential errors in our dataset's numeric label column. Any other sklearn-compatible regression model could be used, such as `LinearRegression` or `RandomForestRegressor` (or you can easily wrap arbitrary custom models to be compatible with the sklearn API)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c2f1ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = HistGradientBoostingRegressor()\n",
    "cl = CleanLearning(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e1b7860",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_issues = cl.find_label_issues(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43bd6c7f",
   "metadata": {},
   "source": [
    "`CleanLearning` internally fits multiple copies of our regression model via cross-validation and bootstrapping in order to compute predictions and uncertainty estimates for the dataset. These are used to identify label issues (i.e. likely corrupted `y`-values).\n",
    "\n",
    "This method returns a Dataframe containing a label quality score (between 0 and 1) for each example in your dataset. Lower scores indicate examples more likely to be mislabeled with an erroneous `y` value. The Dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating its `y`-value appears potentially corrupted). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f407bd69",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_issues.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ab5acf3",
   "metadata": {},
   "source": [
    "We can get the subset of examples flagged with label issues, and also sort by label quality score to find the indices of the 10 most likely mislabeled examples in our regression dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7385336",
   "metadata": {},
   "outputs": [],
   "source": [
    "identified_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
    "lowest_quality_labels = label_issues[\"label_quality\"].argsort()[:10].to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59fc3091",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"cleanlab found {len(identified_issues)} potential label errors in the dataset.\\n\"\n",
    "    f\"Here are indices of the top 10 most likely errors: \\n {lowest_quality_labels}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa2c1fec",
   "metadata": {},
   "source": [
    "Let’s review some of the values most likely to be erroneous. To help us inspect these datapoints, we define a method to print any example from the dataset, together with its given (original) label and the suggested alternative label predicted by your regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00949977",
   "metadata": {},
   "outputs": [],
   "source": [
    "def view_datapoint(index):\n",
    "    given_labels = label_issues[\"given_label\"]\n",
    "    predicted_labels = label_issues[\"predicted_label\"].round(1)\n",
    "    return pd.concat(\n",
    "        [X_train_raw, given_labels, predicted_labels], axis=1\n",
    "    ).iloc[index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c1ae3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "view_datapoint(lowest_quality_labels[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2be7a93",
   "metadata": {},
   "source": [
    "These are very clear errors that cleanlab has identified in this data! Note that the `given_label` does not correctly reflect the final grade that these student should be getting. \n",
    "\n",
    "cleanlab has shortlisted the most likely label errors to speed up your data cleaning process. With this list, you can decide whether to fix these label issues or remove erroneous examples from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9131d82d",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# This cell is hidden from docs.cleanlab.ai \n",
    "\n",
    "label_issues_cl = label_issues.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2761486",
   "metadata": {},
   "source": [
    "## 4. Train a more robust model from noisy labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043bfb52",
   "metadata": {},
   "source": [
    "Fixing the label issues manually may be time-consuming, but cleanlab can filter these noisy examples and train a model on the remaining clean data for you automatically.\n",
    "\n",
    "To establish a baseline, let’s first train and evaluate our original Gradient Boosting model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31c704e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_model = HistGradientBoostingRegressor()  \n",
    "baseline_model.fit(X_train, y_train)\n",
    "\n",
    "preds_og = baseline_model.predict(X_test)\n",
    "r2_og = r2_score(y_test, preds_og)\n",
    "print(f\"r-squared score of original model: {r2_og:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d01f715",
   "metadata": {},
   "source": [
    "Now that we have a baseline, let’s check if using `CleanLearning` improves our test accuracy.\n",
    "\n",
    "`CleanLearning` provides a wrapper that can be applied to any scikit-learn compatible model. The resulting model object can be used in the same manner, but it will now train more robustly if the data has noisy labels.\n",
    "\n",
    "We can use the same `CleanLearning` object defined above, and pass the label issues we already computed into `.fit()` via the `label_issues` argument. This accelerates things; if we did not provide the label issues, then they would be re-estimated via cross-validation. After the issues are estimated, `CleanLearning` simply removes the examples with label issues and retrains your model on the remaining clean data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bcc43db",
   "metadata": {},
   "outputs": [],
   "source": [
    "found_label_issues = cl.get_label_issues()\n",
    "cl.fit(X_train, y_train, label_issues=found_label_issues)\n",
    "\n",
    "preds_cl = cl.predict(X_test)\n",
    "r2_cl = r2_score(y_test, preds_cl)\n",
    "print(f\"r-squared score of cleanlab's model: {r2_cl:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aea51da",
   "metadata": {},
   "source": [
    "We can see that the coefficient of determination (r-squared score) of the test set improved as a result of the data cleaning. Note that this will not always be the case, especially when we are evaluating on test data that are themselves noisy. The best practice is to run cleanlab to identify potential label issues and then manually review them, before blindly trusting any evaluation metrics. In particular, the most effort should be made to ensure high-quality test data, which is supposed to reflect the expected performance of our model during deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "167fca90",
   "metadata": {},
   "source": [
    "## 5. Other ways to find noisy labels in regression datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b4f8e14",
   "metadata": {},
   "source": [
    "The `CleanLearning` workflow above requires a sklearn-compatible model. If your model or data format is not compatible with the requirements for using `CleanLearning`, you can instead run [cross-validation on your regression model to get out-of-sample predictions](https://docs.cleanlab.ai/stable/tutorials/pred_probs_cross_val.html), and then use the `Datalab` audit to estimate label quality scores for each example in your dataset.\n",
    "\n",
    "This approach requires two inputs:\n",
    "\n",
    "- `labels`: numpy array of given labels in the dataset. \n",
    "- `predictions`: numpy array of predictions for each example in the dataset from your favorite model (these should be out-of-sample predictions to get the best results)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7021bd68",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get out-of-sample predictions using cross-validation:\n",
    "model = HistGradientBoostingRegressor()\n",
    "predictions = cross_val_predict(estimator=model, X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d49c990b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cleanlab import Datalab\n",
    "\n",
    "lab = Datalab(\n",
    "    data=train_data.drop(columns=[\"true_final_score\"]),\n",
    "    label_name=\"final_score\",\n",
    "    task=\"regression\",\n",
    ")\n",
    "\n",
    "lab.find_issues(\n",
    "    pred_probs=predictions,\n",
    "    issue_types={\"label\": {}},  # specify we're only interested in label issues here \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbab6fb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_issues = lab.get_issues(\"label\")\n",
    "\n",
    "label_issues.sort_values(\"label_score\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a0db9b2",
   "metadata": {},
   "source": [
    "As before, these label quality scores are continuous values in the range [0,1] where 1 represents a clean label (given label appears correct) and 0 a represents dirty label (given label appears corrupted, i.e. the numeric value may be incorrect). You can sort examples by their label quality scores to inspect the most-likely corrupted datapoints.\n",
    "\n",
    "If possible, we recommend you use `CleanLearning` to wrap your regression model (over providing its pre-computed predictions) for the most accurate label error detection (that properly accounts for aleatoric/epistemic uncertainty in the regression model). To understand how these approaches work, refer to our paper: **[Detecting Errors in Numerical Data via any Regression Model](https://arxiv.org/abs/2305.16583)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b39b8b5",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# This cell is hidden from docs.cleanlab.ai\n",
    "np.random.seed(SEED)  # for reproducibility\n",
    "random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4366346a",
   "metadata": {},
   "source": [
    "You can alternatively provide `features` to `Datalab` instead of pre-computed predictions. These are (preprocessed) numeric dataset covariates, aka independent variables to the regression model (such as neural network embeddings of your raw data).  Internally, this is equivalent to using `CleanLearning` to find label issues if you also possible provide your sklearn-compatible regression model to `Datalab.find_issues`. But you can simultaneously detect many more types of issues in your dataset beyond mislabeling via Datalab (simply drop the `issue_types` argument below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df06525b",
   "metadata": {},
   "outputs": [],
   "source": [
    "lab = Datalab(\n",
    "    data=train_data.drop(columns=[\"true_final_score\"]),\n",
    "    label_name=\"final_score\",\n",
    "    task=\"regression\",\n",
    ")\n",
    "\n",
    "lab.find_issues(\n",
    "    features=X_train,\n",
    "    issue_types={  # Optional drop this to simultaneously detect many types of data/label issues \n",
    "        \"label\": {\n",
    "            # Optional: Specify which type of sklearn-compatible regression model is used to find label errors\n",
    "            \"clean_learning_kwargs\": {\"model\": HistGradientBoostingRegressor()}\n",
    "        }\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05282559",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_issues = lab.get_issues(\"label\")\n",
    "\n",
    "label_issues.sort_values(\"label_score\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1353758",
   "metadata": {},
   "source": [
    "While this tutorial focused on label issues, cleanlab's `Datalab` object can automatically detect many other types of issues in your dataset (outliers, near duplicates, etc).\n",
    "Simply remove the `issue_types` argument from the above call to `Datalab.find_issues()` above and `Datalab` will more comprehensively audit your dataset (a default regression model will be used if you don't specify the model type).\n",
    "Refer to our [Datalab quickstart tutorial](./datalab/datalab_quickstart.html) to learn how to interpret the results (the interpretation remains mostly the same across different types of ML tasks).\n",
    "\n",
    "**Summary:** To detect many types of issues in your regression dataset, we recommend using `Datalab` with provided `features` plus the best regression model you know for your data. If your goal is to train a robust regression model with noisy data rather than detect data/label issues, then use `CleanLearning`. Alternatively, if you don't have a sklearn-compatible regression model or already have pre-computed predictions from the model you'd like to rely on, you can pass these predictions into `Datalab` directly to find issues based on them instead of providing a regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a7a5387",
   "metadata": {},
   "source": [
    "## Spending too much time on data quality?\n",
    "\n",
    "Using this open-source package effectively can require significant ML expertise and experimentation, plus handling detected data issues can be cumbersome.\n",
    "\n",
    "That’s why we built [Cleanlab Studio](https://cleanlab.ai/blog/data-centric-ai/) -- an automated platform to find **and fix** issues in your dataset, 100x faster and more accurately.  Cleanlab Studio automatically runs optimized data quality algorithms from this package on top of cutting-edge AutoML & Foundation models fit to your data, and helps you fix detected issues via a smart data correction interface. [Try it](https://cleanlab.ai/) for free!\n",
    "\n",
    "<p align=\"center\">\n",
    "  <img src=\"https://raw.githubusercontent.com/cleanlab/assets/master/cleanlab/ml-with-cleanlab-studio.png\" alt=\"The modern AI pipeline automated with Cleanlab Studio\">\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95531cda",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from cleanlab.regression.rank import get_label_quality_scores\n",
    "\n",
    "if r2_cl <= r2_og:\n",
    "    raise ValueError(\"CleanLearning did not improve r2 score\")\n",
    "\n",
    "label_quality_score_cl = label_issues_cl[\"label_quality\"]\n",
    "label_quality_scores_residual = get_label_quality_scores(labels=y_train, predictions=predictions, method=\"residual\")\n",
    "\n",
    "label_quality_scores = get_label_quality_scores(labels=y_train, predictions=predictions)\n",
    "\n",
    "auc_outre = roc_auc_score(errors_mask, 1 - label_quality_scores)\n",
    "auc_cl = roc_auc_score(errors_mask, 1 - label_quality_score_cl)\n",
    "auc_residual = roc_auc_score(errors_mask, 1 - label_quality_scores_residual)\n",
    "\n",
    "if auc_outre <= 0.5 or auc_cl <= 0.5:\n",
    "    raise ValueError(\"Label quality scores did not perform well enough\")\n",
    "\n",
    "if auc_outre <= auc_residual:\n",
    "    raise ValueError(\"Outre label quality scores did not outperform alternative scores\")\n",
    "    \n",
    "if auc_cl <= auc_residual:\n",
    "    raise ValueError(\"CL label quality scores did not outperform alternative scores\")\n",
    "\n",
    "# Test that CleanLearning label issues and Datalab label issues match\n",
    "pd.testing.assert_frame_equal(\n",
    "    # CleanLearning DataFrame\n",
    "    label_issues_cl.rename(columns={\"label_quality\": \"label_score\"}), \n",
    "    # Datalab DataFrame\n",
    "    label_issues,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
